Feature Extraction for Multiple Kernel Learning

Multiple Kernel Learning (MKL) synthesizes a single kernel from a set of multiple kernels for use in a support vector machine. We propose that MKL be preceded by feature extraction. Given a set of kernels and a vector y of class labels, Multiple Kernel Basis Extraction (MKBE) constructs orthogonal vectors {v1, . . . , vm} whose corresponding kernels, {v1v 1 , . . . , vmv m}, are maximally aligned with yy . Each of these vectors maximizes a Rayleigh quotient with respect to one of the given kernels, subject to orthogonality constraints. Standard MKL techniques can then be applied to the extracted set of rank-one kernels. Theoretical considerations suggest that preliminary feature extraction may improve classifier performance. Examples illustrate that the improvement can be substantial.